Greedy-layer pruning: Speeding up transformer models for natural language processing

Speeding Up FastICA by Mixture Random Pruning

We study and derive a method to speed up kurtosis-based FastICA in presence of information redundancy, i.e., for large samples. It consists in randomly decimating the data set as more as possible while preserving the quality of the reconstructed signals. By performing an analysis of the kurtosis estimator, we find the maximum reduction rate which guarantees a narrow confidence interval of such ...

Connectionist Models for Natural Language Processing Program

The scientific adequacy of models based on a small number of coarse-grained primitives (e.g. conceptual dependency), popular in AI during the 70's, has been called into question and substantially replaced by a current emphasis in much of computational linguistics on lexicalist models (i.e., ones which use words for representing concepts or meanings). However, few people can doubt that words are...

Maximum Entropy Models for Natural Language Processing

TFLEX: Speeding Up Deep Parsing with Strategic Pruning

This paper presents a method for speeding up a deep parser through backbone extraction and pruning based on CFG ambiguity packing.1 The TRIPS grammar is a wide-coverage grammar for deep natural language understanding in dialogue, utilized in 6 different application domains, and with high coverage and sentence-level accuracy on human-human task-oriented dialogue corpora (Dzikovska, 2004). The TR...

Speeding up LFG Parsing Using C-Structure Pruning

In this paper we present a method for greatly reducing parse times in LFG parsing, while at the same time maintaining parse accuracy. We evaluate the methodology on data from English, German and Norwegian and show that the same patterns hold across languages. We achieve a speedup of 67% on the English data and 49% on the German data. On a small amount of data for Norwegian, we achieve a speedup...